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I. INTRODUCTION 



A. PROBLEM STATEMENT 

The main question that this thesis will attempt to answer is whether or not there is 
a correlation between performance in a real world building interior navigation task and a 
virtual world building interior navigation task. The thesis will analyze and compare an 
individual’s performance in both environments and draw conclusions about the measures 
we have been using by looking at the results of their performance in these two 
environments. The research intends to question a frequently implied assumption that the 
same performance measures are appropriate in virtual and real world environments. If a 
correlation is found, further research might investigate if it’s also valid in other types of 
environments (such as forests and other natural spaces). If disproved in the building 
environment, the relationship can still be investigated for natural environments. The lack 
of a correlation will suggest the causes to be either that the sensitivity of the measures 
was too low or that the characteristics of the virtual environments interface or fidelity 
inhibited natural spatial knowledge acquisition. 

B. MOTIVATION 

1. Applications for Spatial Knowledge Acquisition 

Spatial knowledge acquisition is used in a wide range of jobs and activities. In our 
daily lives we happen to use our spatial knowledge acquisition capability to find an 
address, to go to an office in an unfamiliar building, or to find the quickest route in an 
emergency. When we think of certain jobs like policemen, firemen, and paramedics, their 
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need for the possession of this kind of knowledge becomes more clear. A policeman who 
gets into a building for the first time and attempts to take a certain route in the building 
will have to acquire the information about the routes, landmarks and general map of the 
building. Similarly, a fireman will need acquisition of this spatial knowledge to 
maneuver in an environment where there are flames and smoke. A taxi driver needs 
spatial knowledge to take the shortest route in a city and not cause any delay in his 
customer’s schedule. 

When it comes to the military, the acquisition of spatial knowledge gets even 
more important. Spatial knowledge acquisition is the main issue for the navigation of 
dismounted soldiers. Spatial knowledge is not limited to soldiers, but must also be 
acquired by tank drivers, pilots, and members of any maneuver unit. Knowledge of the 
environment is the basic knowledge needed to accomplish any of the military missions 
whether it is to attack, retreat, or reconnoiter. Even in defense, when there is no 
movement, acquisition of spatial features of the environment is essential. 

So, spatial awareness of the environment is a requirement for the accomplishment 
of even relatively static tasks. And it gets even more important when movement is 
involved in the specific task. 

2. Army and DOD Relevance 

Traditional methods of training and mission preparation may be enhanced by 
virtual environment (VE) augmentation. Mock-ups or other representations of the training 
environments were used as training methods, but these had many disadvantages. 
Considering only the costs associated, they were expensive, and it took a lot of time to 
build them. Furthermore, they used to occupy a large area and it was not so easy to 
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modify them according to the changes in the structure of the mission. The likely need for 
the replacement of these methods by VE technology has generated the issue of using 
virtual environments in training or mission preparation. 

The representation of the real environment by a virtual environment has its own 
assumptions. One major postulate is that the performance measures in a virtual world 
would be equally sensitive to performance in the real world. Spatial knowledge 
acquisition in the virtual world has been evaluated with the same criteria which is used in 
the real world. However, there is no concrete basis for this assumption. The army, using 
virtual environment in its navigation studies, would like to know if there is a positive 
correlation between the measures used in these two different environments. 

Additionally, army recruiters need to assess the capabilities of applicants for the 
army. Since the spatial knowledge has been proved as having significant importance in 
Army activities, it would be useful if the spatial knowledge ability of recruits could be 
assessed in a virtual environment navigation task. When we use such a task, for an 
accurate evaluation we want to make sure that the measures we use for the virtual 
environment do represent the same measures in the real world. In addition, we would like 
to know if there are some measures which do not necessarily apply to the real world and 
vice versa. All these assessment efforts will require a deep analysis of the measures used 
in virtual environments. 

3. Existing Shortcomings of Research 

Computer based training is gaining more popularity every day. However, except 
for the cost savings, there is not much proof of the advantage gained from using computer 
based systems. Furthermore, when these systems are used, we want to make sure that our 
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evaluation of the capabilities is a realistic evaluation, and it assesses the individual 
capability on land navigation. So far, there has not been a study specifically investigating 
this correlation; rather the measures of real and virtual worlds have always been assumed 
to be related. Furthermore, while this study is not a transfer study, it does contribute to the 
transfer study literature by evaluating an assumption used in this kind of study. 

C. THESIS ORGANIZATION 

This thesis is organized as follows. Chapter I is the introduction to the area, 
describes the problem that the thesis tries to solve, gives an explanation of the motivation 
points for the thesis, and the organization of the thesis. Chapter II covers the background 
and previous work in the area. Mainly, the second chapter tries to introduce the existing 
measures of performance in land navigation, compares them, and identifies the main 
measures to be used in this study. Chapter III discusses the virtual and real environments 
used in this thesis. It also gives a detailed description of the experiments used for the 
study and the methodology of the experiments. Chapter IV analyzes the data collected 
from the two experiments, and discusses the results. Chapter V draws conclusions from 
the results of the experiments and discusses the impact of the conclusions to the studies in 
the area. Chapter V also lays out future research lines in this subject. 

The thesis has several appendices which show the instructions to the participants, 
maps of the buildings used for the study, briefing forms, and the data sheets collected 
from the study. The appendices will help the reader to understand the nature of the 
experiments better, and will help researchers trying to replicate this work. 
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II. BACKGROUND AND PREVIOUS WORK 



A. GENERAL DESCRIPTION OF THE CHAPTER 

This chapter is a literature review on performance measures in a land navigation 
task. After an introduction to the subject, the author describes the concept of spatial 
knowledge with its three components: Landmark Knowledge, Route Knowledge and 
Survey Knowledge. Throughout the review of the literature, the author handles the 
measures of these three levels separately, and examines the advantages and disadvantages 
of each technique in measuring the performance level. Finally, four performance 
measures that can be used in a study to understand the correlation of the measures in 
virtual and real environments are determined considering the efficiency and applicability 
of the measures: landmark recognition, route replication (number of wrong turns), 
pointing, and map sketching. 

B. INTRODUCTION TO THE BACKGROUND AND PREVIOUS WORK 

Spatial knowledge acquisition has always been an integral part of navigation 
related studies. With the improvement of technology, researchers gained the capability of 
testing spatial ability in a virtual world as well. Researchers have been using different 
measures to assess spatial ability either in a real world or a virtual world land navigation 
task. However, there is no concrete evidence that supports the idea that the performance 
measures used in a real world task should necessarily correlate with the performance 
measures used in a virtual task. Anybody doing a study on land navigation of real and 
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virtual environments should naturally be aware of the measures that are pretty much 
applicable in a real world, but do not correlate to the virtual world and vice versa. This 
chapter, firstly examines the issue of spatial awareness, and reviews the performance 
measures that are commonly used in navigation tasks. Finally, from the most common 
performance measures, the author determines the five best measures for investigating real 
and virtual world performance. 

C. SPATIAL KNOWLEDGE 

In the widely accepted theorization, there are three levels of spatial knowledge: 
landmark knowledge, route knowledge and survey knowledge. (SIEGEL & WHITE, 75) 

1. Landmark Knowledge 

Landmark knowledge is defined as the ability of the navigator to recognize 
distinctive features or locations at specific locations in the navigated area. When an 
outdoor navigation task is considered, landmark knowledge is the ability to recognize the 
features in an environment, such as a hilltop, a bridge, an intersection, or a building. 
Likewise, in an indoor navigation task, landmark knowledge would be the ability to 
recognize places where some certain objects are located. Although landmark knowledge 
can be gained through photographs or sketches of the area, generally the best way to gain 
landmark knowledge is direct exposure to the area. A performance success in landmark 
knowledge is assessed by evaluating the ability to recognize distinct locations or unique 
objects in the area, but not by knowing where they are. 
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2 . 



Route Knowledge 



Route Knowledge is defined as the ability of the navigator to navigate along a 
route or path between landmarks or distant locations [GOLLEDGE 91]. It expands 
landmark knowledge to a larger, more complicated arrangement of linking those objects 
or locations by a path or route. Route knowledge can be gained by repeated exposure to 
the environment, by maps or through a simulated exposure to the environment [GOLDIN 
82]. Route knowledge is based on an egocentric viewpoint. Evaluating the ability to 
move from one landmark to another along a prescribed path assesses route knowledge. It 
requires the knowledge of the order between landmarks as well. The level of route 
knowledge gives no indication about the navigator’s ability to find or follow alternate 
routes between landmarks. 

3. Survey Knowledge 

Finally, the highest level of spatial knowledge, survey (or configurational) 
knowledge represents a map-like or top down mental encoding of the environment and is 
based on a geocentric viewpoint. Survey knowledge is considered the most valuable part 
of the navigator’s spatial knowledge. Because a person with survey knowledge has a 
mental map of the area, and can easily figure out alternate ways in that area. Sometimes, 
it’s even a basis for making a distinction between a route learner and a navigator [HUNT 
& WALLER 99]. In survey knowledge, the navigator not only recognizes the specific 
locations or landmarks but also has a mental representation of them and can accurately 
place them in the environment even if he or she can not see them. The navigator can also 
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traverse through the area without having to pre-plan the routes, because they know the 
area and they can devise new routes between landmarks [BANKER 97], 

D. PERFORMANCE MEASURES 

A review of the literature on spatial knowledge will result in the selection of five 
main performance measures to be tested for future work on the correlation between the 
measures in real and virtual worlds. These measures should be the measures pointing to 
the three main components of spatial knowledge. It’s also expected that the measures 
identified in the review can be related to more than one component of spatial knowledge. 



PERFORMANCE MEASURES 


SPATIAL KNOWLEDGE TYPE 


Landmark Recognition 


Landmark 


Order of Landmarks 


Landmark, route 


Number of Wrong Turns 


Route, survey 


Verbal Recollection of Spatial Experience 


Landmark, route, survey 


Number of People Losing Their Way 


Route, survey 


Sketch Map 


Survey 


Whiteboard Test 


Survey 


Projective Convergence 


Survey 


Map Placement 


Survey 


Pointing Test 


Survey 



Table 2.1 Spatial Knowledge type for each Performance Measure 
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1 . 



Measures of Landmark Knowledge 



Landmark recognition is almost the only measure in evaluating a navigator’s 
landmark knowledge. It is assessed by asking observers to recognize or recall the 
landmarks that they have seen along the route [EVANS, 1980]. Recognition and recall 
have been evaluated through different types of tests. One way to test landmark 
recognition is getting the pictures of landmarks existing along the route, mixing these 
with some irrelevant landmark pictures, and asking the observer to choose the landmarks 
that he or she has seen throughout the navigated route. Assigning a positive weight to the 
correct and negative weight to the false response forms the scale for that particular 
individual’s landmark recognition score. 

2. Measures of Route Knowledge 

There are several measures of route knowledge. One of these is the verbal 
recollection of spatial experiences [LYNCH, I960]. By using the verbal recollection of 
experiences, it’s possible to determine the way the navigator connects the landmarks in 
his or her memory. This recollection of experiences in fact shows the mental route that 
the navigator thinks he or she has followed. This measure is not an easily quantifiable 
measure, but will give a good idea about a certain individual’s route knowledge. 

Other measures used to evaluate the route knowledge of the navigator are the 
number of wrong turns and route traversal times [STREETER, et al.]. This is a rather 
quantifiable measure when compared to verbal recollection. A participant having his or 
her second traversal in the environment after a first exposure (or map study, VE 
exposure) is observed to assess the wrong turns he or she makes and the time he or she 
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spends in the virtual environment. Obviously, time will differ in the real and virtual 
environments, since the factors affecting the time in a real environment are not 
necessarily effective in a virtual environment (e.g. travel methods). However, these 
factors don’t necessarily have a relation with spatial ability either. To clarify this, we can 
think about the rate of movement in navigation. A fast walker in a virtual environment 
moves at the same rate as a slow walker, and walking fast or slow does not necessarily 
have any relation with a person’s spatial ability. 

In larger group studies, the number of people who lose their way [BEST, 1969] is 
also a measure used to determine the route knowledge acquisition for that environment. 
However, in a study to determine the spatial ability or performance level of each 
participant individually, this measure cannot be used. However whether or not one person 
is lost would be a measure for that individual’s performance. When we think about the 
percentage of people getting lost in an environment, that would not be an appropriate 
measure for a particular person in that group. 

One other measure for route knowledge is the arrangement of photos of route 
segments and landmarks in their correct order [EVANS, SKORPANICH, GARLING, 
BRYANT & BRESOLIN, 1984], As it is expressed in previous pages, some of the 
measures being mentioned in this chapter can evaluate different levels of spatial 
knowledge at the same time. The last mentioned measure is a measure that assesses 
landmark knowledge as well as route knowledge. 
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3. Measures of Survey Knowledge 

The highest level of spatial knowledge, survey (or configurational) knowledge, is 
the main factor that makes the difference between a route learner and a navigator. 
However, survey knowledge is difficult to measure. The mental map in each navigator’s 
mind cannot be quantified easily. 

Generally, configuration knowledge is measured by asking the participants to 
draw a sketch map of the navigated environment [LYNCH, 1960; APPLEYARD, 1970]. 
However, the maps drawn are difficult to score, and may underestimate the knowledge 
acquired [SIEGEL, 1981]. 

A measure used by Goerger was the “Whiteboard Test” which measures the 
exocentric survey knowledge of the participant [GOERGER, 1998]. This measure was 
able to overcome the effect of drawing abilities between different individuals. 

Another measure, the projective convergence technique [SIEGEL, 1981], requires 
participants to estimate the bearing and distance to landmarks obscured from view from 
three different sighting locations. A technique very similar to this one is the pointing 
technique which simply calculates the angular error between the pointed direction and the 
actual direction of an unseen object in the environment. 
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E. SELECTED MEASURES 



A consideration of the measures mentioned in previous sections requires the 
selection of the most efficient ones among them. Without any doubt, a landmark 
recognition task, which is the only exclusive way to determine landmark knowledge 
should be included in a performance measure study. The correct order of the landmarks 
will provide information about the navigator’s route knowledge. A route replication task 
to get a measures of the number of wrong turns will provide a good basis on the person’s 
route knowledge. Then, for the survey knowledge, a pointing task and a sketch map will 
be used. The sketch map provides input on the route knowledge as well. 

F. SUMMARY OF THE CHAPTER 

In summary, the human performance measures used in either virtual or real 
environments is a broad area. Spatial knowledge itself has three components, which have 
their own effect on environmental knowledge acquisition. Some of the measures used in 
the previous studies are applicable in certain cases, but not in some others. Likewise, 
these measures might be evaluating just one aspect of spatial knowledge or different 
aspects of spatial knowledge. Therefore each of the measures has its own efficiency and 
applicability level. 

For a further study of land navigation performance level correlation analysis, the 
author has selected five performance measures as the most efficient applicable ones due 
to their advantages or disadvantages. These measures are, landmark recognition, 
landmark sequence, route replication (number of wrong turns), pointing, and map 
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placement. Table 2.2 shows the list of the selected measures and the spatial knowledge 
types they measure. 



SELECTED PERFORMANCE MEASURES 


SPATIAL KNOWLEDGE TYPE 


Landmark Recognition 


Landmark 


Order of Landmarks 


Landmark, route 


Number of Wrong Turns 


Route, survey 


Map Placement 


Survey 


Pointing Test 


Survey 



Table 2.2 Selected Measures and the Spatial Knowledge Type they measure 



Any study using human performance measures in a virtual or real world 
navigation task should expose and address the power of the measures to evaluate the 
performance levels, and the applicability of the measure for that particular study. Even 
indoor and outdoor navigation tasks might require quite different types of these measures. 
Furthermore, none of the components of spatial knowledge mentioned in this study 
should be neglected in an evaluation task. The most important measures, which are at the 
same time the most commonly used ones, should be examined for a comparison study. 
Thus, the results of further studies should be evaluated on a concrete basis. 
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III. METHODOLOGY 



A. EXPERIMENT OVERVIEW 

An experiment was conducted in order to determine whether or not the 
performance measures in real and virtual worlds are correlated. The succeeding sections 
will describe the tools, phases and procedure of the experiment in more detail. This 
section will provide an overview for the experiment conducted in two different 
environments. The general sequence of the experiment was the in-briefing, model 
familiarization (VE only), walking through the course, route replication, landmark 
recognition, free exploration of the environment, finding alternate routes, map placement 
and debriefing. 

Upon the arrival of the participant, the research monitor read the basic in briefing, 
and they filled out and signed basic consent forms. The in brief is shown in Appendix C. 
The consent forms are in Appendix D. 

In the virtual environment condition, the subject was initially told how to operate 
the Flybox. Forward and backward movement switches and the usage of the joystick in 
order to face different directions or look down or up were also explained in the 
familiarization phase. A common error detected in the pilot experiment and previous 
attempts was trying to get a rolling effect. Each participant was told that the Flybox does 
not have such a function. After the verbal instructions of the Flybox, a trial model was 
started. In this model, the participant took a very short walk in the building, and this route 
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was a part of the building excluded from the experimental building. In the trial model, the 
participant started from a specified point and moved as instructed by the research 
monitor. Throughout the trial he was allowed to move forward or backward, make 
necessary turns, in some narrow aisles, go down and up the stairs. All the movements in 
the trial model were as difficult as the real model. In the trial model, the participants 
replicated the route and were given pointing tests. Each participant had an idea of what he 
was supposed to do in the real experiment. There was no time limit for the trial model, 
but all participants reported that they felt comfortable with Flybox at the end of the trial 
task. 

After the familiarization, each participant started to walk on the predetermined 
route according to the instructions given by the research monitor. At a certain point, 
(referred as point A) the participant was stopped and asked to face the starting point. In 
the virtual model, the degree was taken by a keystroke, and in the real building a pointing 
tool was provided to allow the participant point to the starting place while taking a 
measurement. (Figure After the first pointing test, the participant was asked to walk 
according to the instructions until the end point. At the end point, two more pointing tests 
were conducted. In these two tests, the participant was asked to point to the starting place 
and to point A. 

Upon the completion of the route, the participant was told to follow the same 
route in reverse order to the starting point. This was the route replication task for the 
experiment. In the route replication task, the participant was allowed to look around to 
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determine the route to take, but whenever he decided on a route and started to walk it, if 



the direction was incorrect, he was directed to the correct path. In each of these cases the 
wrong turn was recorded by the research monitor. By making the required turns, the 
participant reached the starting point. At this point, a fourth pointing task was conducted. 
The participant was asked to point to the end point. 

When the route replication task was completed, a landmark recognition test was 
conducted. In this task, the participant was shown ten different landmarks, some of which 
had existed in the original route. For this test, color printed pictures of the landmarks 
were used and the number of correct landmarks were recorded. When the participant 
selected the landmarks he had recognized, he was asked to put them into correct order 
from the starting point to the end point. The order of the landmarks for each participant 
was also recorded for further analysis. 

Upon the completion of the landmark recognition test, each participant was asked 
to explore the environment himself for 10 minutes. For the virtual building, at the end of 
the allowed time, the research monitor exited the program wherever the participant was. 
For the real building, in the last ninety seconds, the research monitor led the participant to 
the starting point in order to make sure that everybody uses the same amount of time. 

After the free exploration of the environment, a task of finding alternate routes 
was conducted. The participant was told to take the exact route he took from the starting 
point to the end point. At a certain place in this route he was given a scenario. In the 
scenario, a stairway was blocked due to construction and the participant was asked to find 
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an alternate route he thinks will reach the end point. Then he was instructed to take that 
alternate route. Whenever he realizes that his alternate route was not correct, he was 
asked to determine a new alternate route until he reaches the end point. In each of these, 
his level of confidence was also recorded. 

When the participant completed the task of finding alternate routes, he was given 
the landmarks that he had recognized in small pictures. Then, he was asked to locate them 
at the appropriate points on a given map. The map was a rough design of the building just 
showing the outer lines and the starting point. The places of the landmarks were recorded 
with the map. 

After the completion of the experiments in both environments, the subjects were 
debriefed about the experiment. They were thanked and reminded not to talk to anyone 
about the specifics of the experiment. 

B. EXPERIMENTAL DESIGN 

The participants were divided into two groups. The first group experienced the 
real building first, and then the virtual building. The second group had the reverse order. 

1. Participants 

The subjects for this experiment consisted of 10 male individuals ranging in age 
from 24 to 27 with an average of 25. All were active duty Turkish Army officers. None of 
the subjects was colorblind. Since a certain level of experience in navigation was not 
required in this experiment, they ranged in different levels of experience and spatial 
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ability. None of the subjects had prior knowledge of the testing area. Data was collected 
from 28 June 2000 to 17 July 2000. 

2. Order of Test Environments 

The participants were tested in two groups. The first group participated first in the 
virtual environment, then in the real world. The second group followed the reverse order. 
However, this should not be taken as two different treatments. The reason for the order of 
exposure to the two different environments was not to test the difference. We wanted to 
avoid errors that could be caused by a learning effect. 

Performance in landmark recognition, landmark sequencing, route replication, 
pointing, and map placement were the five performance measures used in this study. 

C. APPARATUS 

1. Test Environments 

Two environments were used for the experiment. One of them was the Middle 
East School at the Defense Language Institute in Monterey, CA (See Figure 3.1). The 
other was a modified version of Herrmann Hall at the Naval Postgraduate School, 
Monterey, CA. 

We never invented for the two environments to be exactly the same, because this 
could cause a learning effect. However, so as not to have a completely different level of 
performance in these two environments, we looked for similar properties in these two 
buildings. 
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Figure 3.1 Middle East School in DLI 



These two buildings had similarities which made it possible to think of a 
correlation between the performance measures in these two environments. The first 
similarity was the complexity level of the two buildings as defined by the features that 
make the navigation more difficult. There were some features shared by both buildings. 
One feature was the location of the stairways. There were many stairways in different 
parts of both buildings. A second feature was the presence of partial floors in these 
buildings. The Herrmann Hall model has a mezzanine level. In the Middle East School, 
there were three buildings connected to each other, and the connection to a different 
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building was provided by a passage off the halfway of the stairs. So, for example, the 
second floor of one building was between the first and second floors of another building. 
This made it possible to lose the concept of vertical location throughout navigation inside 
both buildings. Both of the buildings were closed environments and it was not possible to 
get much reference from the outside. Neither the Herrmann Hall model nor the Middle 
East School has sufficient windows to view outside. Since the classrooms in the Middle 
East School were not accessible throughout the experiment, it was not possible to get a 
sense of location by looking outside. In Herrmann Hall, there were rare windows through 
which to view outside. 

2. Virtual Model 

The virtual model is a modified form of the Herrmann Hall model created by John 
Locke at the Naval Postgraduate School. The model was created in Creator, by Multi gen 
Paradigm and the motion model for it was developed by Brian Christianson and Andrew 
Kimsey at NPS. The model itself runs on an SGI machine with four 194 MHZ IP25 
processors and 384 Mbytes main memory size. 

The interface uses Flybox (see figure 3.2) as the interaction device. Flybox allows 
the user to face right or left, or look up or down. The speed of walking was also 
designated by the user by moving a lever into forward and backward positions. When the 
lever was kept in the center, there was no forward or backward motion. 
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Figure 3.2 Flybox 

The table 3. 1 shows the effects of Flybox. 
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FLYBOX OPERATION 


| 

FUNCTION 

1 


Left Switch “Forward” 


Increase forward speed 


Left Switch “Backward” 


Increase backward speed 


Left Switch “Center” 


Stop 


Move Joystick to the “Left” 

1 1 


Turn the body to the left 

■ 


Move Joystick to the “Right” 


Turn the body to the right 


Move Joystick “Forward” 


Look Down 


Move Joystick “Backward” 


Look Up 



Table 3.1 Functions of the Flybox 



For the visual display, a three-screen configuration with 103 - degree FOV was 
used. (See figure 3.3) These three 40 inch screens were set up exactly 67 inches from the 
subject to ensure that 103 degrees were provided. The measurement for the visual display 
was based on the previous study made by Goerger [GOER 98]. 
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Figure 3.3 Three screen visual display 

In summary, the model was set to make sure that the participants were provided 
the comfort and ability they needed to navigate in a virtual building. 

3. Route 

The route for the task was a route including the same level of vertical motion and 
approximately the same number of turns for both real and virtual environments. 

In the Herrmann Hall model, there were 9 turns throughout the navigation, and 
the user was navigating in three floors of the building. One of the floors was a mezzanine, 
which had an entrance halfway up the stairway. The route for Herrmann Hall is shown in 
Figure 3.3. The oval shapes in the graph indicate the stairways. 
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Figure 3.4 Plan View of the Route for the Hemnann Hall model 

The real building route was similar to the Herrmann Hall model with respect to 
the number of turns and floors in the building. The route in the Middle East School had 8 
turns and three floors throughout the navigation. This building consisted of three distinct 
parts connected by stairways. (The stairways were connecting one floor to the other. See 
figure 3.6) Consequently, the second floor of one building was vertically located at a level 
between the first and second levels of the other building. This feature of the real building 
was similar to the mezzanine of Herrmann Hall. A graph of the route for the real building 



25 




is in Figure 3.4. In this figure the oval shapes indicate the stairways, as in the previous 
figure. The routes themselves are seen in Appendix E. 




Figure 3.5 Plan View of the Route for the Real Building 
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Figure 3.6 Stairways in DU 



D. PROTOCOL 

1. Walking Through the Course 

The first task was to walk through the course upon the instructions of the research 
monitor. From the starting point, the subject was asked to walk straight ahead or turn a 
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direction by the research monitor. The walking speed of the subject was controlled by the 
research monitor, and he was warned if he either walked too slowly or too fast. The 
normal walking speed was determined as the speed of the monitor, and the subject was 
asked to keep that rate. 

In the virtual environment, the speed of the participant was limited with the 
motion model, so the participant could not walk faster than a certain level. In addition to 
this, to prevent slower walking, the participants were verbally warned not to walk slower 
than a walking rate. 

In the real building, since the participant and the research monitor were walking 
together, the participant was asked to keep pace with the research monitor’s walking 
speed. This made it possible for each participant to have equal time for observation. 

2. Route Replication Task 

Once the subject w'alked from the starting point to the end point upon the 
instructions of the research monitor, a route replication test was started from the end 
point. At this point, the subject w'as asked to take the same route back to the starting 
point. In the route replication, although the subject was allowed to look around to make 
decisions on the necessary turn, once he decided and started to proceed on a route, that 
decision w r as recorded. If the subject has chosen a wrong way and started on that route, he 
w f as corrected, and that turn was recorded as a wrong turn. Each time the subject made 
such a wrong turn, the wrong turn was recorded. In this way, the subject w'alked from the 
end point to the starting point in the same path. 
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3. Landmark Recognition Test 

Upon the completion of route replication, the subjects were given ten pictures of 
certain landmarks and were asked to select the ones they recognized. The landmarks used 
in the experiment are in appendix F. Once the subject chose the landmarks that he had 
recognized, he was asked to put them into the order that he had seen them from the 
starting to the end point. The number of correct landmarks that the subject recognized and 
the order of landmarks were recorded by the research monitor. 

4. Pointing Tests 

Throughout the first route from starting point to the end point, and at the end of 
the route replication, a series of pointing tests were conducted. The first pointing test was 
conducted halfway from the starting point to the end point. This place was referred to as 
point A throughout the experiment. At point A, the subject was asked to point the starting 
point. 

In the virtual environment, pointing was done by facing that direction, at which 
time, the research monitor recorded the location and the bearing of the subject to a file. 
Two other pointing tests were conducted at the end point. In these test, the subject was 
asked to point to the starting point and point to A from the end point. Besides these, at the 
end of the route replication test, the subject was asked to point to the end point from the 
starting point. This was the fourth and last pointing test for the experiment. 

In the real building, this test was conducted by directing a paper arrow on a board 
to the starting point. At certain points of the building, rather than facing a direction, the 
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participant was asked to direct the arrow on the paper to the location asked. The angular 
degree was recorded at the time of the test conduction. 

Pointing test results were used as a measure of survey knowledge of that subject. 
The degree of error in these tests were used as a clue to understand the survey knowledge 
of the participant in that environment. 

5. Free Exploration 

In the free exploration phase, the participant was allowed to explore the 
environment himself for ten minutes. Throughout this phase, the research monitor walked 
with the participant, and recorded the routes he took. This w r as done, so that if a specific 
route that can affect the results was taken, this could be noted. 
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Figure 3.7 Free Exploration 



In the virtual model, when 
monitor exited the program simply 



the participant used the allowed time, the research 
by hitting the escape key. This had been told to the 



31 




participant prior to the free exploration task. Thus, the allowed time for each participant 
was same in the virtual environment. 

However, it would not be this simple in the real world. In the real world, some 
participants could be close to the starting point at the end of the allowed time, and some 
could be far. This could result in extra time for the participants who are in the farthest 
location to the starting point. We wanted to avoid such a difference between the 
participants. For this reason, at the last 90 seconds of the free exploration, the research 
monitor evaluated how far the participant is from the starting point, and led the 
participant to the entrance of the building where he started. This was done by leading the 
participants via the shortest route, so the same amount of time would be used by each 
participant. Throughout the experiment, each participant was warned about the remaining 
time in one minute intervals. 

6. Alternate Routes 

After the free exploration of the environment, the subjects were asked to walk 
from the starting point to the end point in the same route they took in the first task. When 
they reached a certain point on this route they were given a scenario. According to the 
scenario, a stairway was blocked due to a construction in the building, and the subjects 
were asked to think about alternate routes to reach the end point. Before taking an 
alternate route, the subjects were asked to describe the route and their description of the 
alternate route was recorded by the research monitor. Besides describing their route, they 
were also asked to say the level of confidence they felt for this route. The confidence 
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level for each alternate route was also recorded. The confidence level was a number 



between 1 and 9 from the least confidence to the highest. When the subject reached a 
point where he saw that his alternate route does not go to the end point, he was asked to 
describe a new alternate route and rate his confidence level for this new alternate route. 
Each of these routes and the confidence levels were recorded until the subject found his 
way to the end point. 

7. Map Placement 

Upon the completion of all tasks in the experiment, the subject was asked to place 
the landmarks he had seen throughout the navigation on a rough map of the building. The 
rough map used for this test showed only the outer line of the building and the starting 
point. The subjects were given small pictures of the landmarks that they had seen, and 
were asked to place them into proper locations of the building. The maps that they had 
built this way were recorded as another measure of survey knowledge. Once the exact 
points they had located were recorded, these maps were compared to a map which has 1/2 
inch diameter (corresponding to 9 feet tolerance) circles in the places of the landmarks. 
The difference between the subject’s landmark and that circle was taken as a measure. 
We did not require exact precision. If the subject was not able to remember that 
landmark, he was penalized by the biggest error made by any subject for that specific 
landmark. 

This test should not be confused with the whiteboard test used by previous 
researchers (GOERGER, 1998). In the whiteboard test, a blank white board was used, but 
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the map provided in this experiment used a rough design of the building. Although the 
rooms and the details of the building were not provided in this map, the walls and the 
entrance were indicated, so the participants had a better idea of where to locate the 
landmarks. 
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IV. ANALYSIS 



A. RESULTS 
1. General Information 

The experiment is designed to test a primary hypothesis concerning the correlation 
between the navigational performance of a particular participant in a virtual and real 
environment. To determine overall performance, participants were evaluated on their 
landmark, route and survey knowledge of the environment, while going through a series 
of tasks in the target environments. 

a. Primary Hypothesis 

A participant’s navigation performance in a real world environment will 
be correlated to their performance in a similar but distinct virtual environment. 

b. Sub Hypotheses 

1 . Landmark Knowledge 

a) A participant who has the spatial ability of keeping landmarks in his 

memory in a virtual environment, will have the spatial ability of keeping 

landmarks in his memory in a real world environment. 

2. Route Knowledge 

a) A participant who can recall the order of the landmarks he had seen in a 
virtual environment navigation task, will be able to recall the landmarks he had seen in a 
real world navigation task. 
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b) A participant who can form a route of a course in a virtual environment 
will also be able to form a route of a similar course in the real world environment. 

3. Survey Knowledge 

a) Given a certain time to get familiar with the environment, a navigator 
who can point to unseen objects accurately in a virtual environment will also be able to 
point to unseen objects accurately in a real world environment, after being exposed to that 
real environment for an equal period of time. 

b) A participant who can form a mental map of a virtual environment that 
he or she has learned can also form the mental map of a real environment after being 
exposed to that real environment for a period of time. 

2. Power Analysis 

The results of the experiment are presented as scatter plots and histograms. The 
primary analysis was based on the correlation between the virtual and real treatment 
groups, but other possible factors were analyzed and discussed in Section B. An a value 
of 0.1 was used to determine significance. A post hoc power analysis for large effect, a = 
0.1 was made. The resulting power (1 - P) was 0.5102. Due to the low sample size, it 
may be incorrect to draw conclusions based solely on the failure to determine a positive 
correlation. 

3. Normalization of Data 

Most of the measurements used for analysis in this experiment occurred over time 
and distance. Some participants were not able to locate certain landmarks just because 
they had not remembered the landmarks themselves. To make participant data 
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comparable, measurements were modified with penalties. This allowed us to place each 
participant’s data in a rational format for correlative analysis. 

4. Landmark Knowledge 

Landmark knowledge is assessed by analyzing the landmarks recognized by the 
participant and sequence of landmarks formed by the participant. 

a. Number of Landmarks Recognized 

After going through the environment for a certain period of time and a 
certain distance, each participant was shown ten landmarks, six of which had existed in 
the environment. From these landmarks, the participants were asked to select the ones 
that they had seen throughout the navigation exercise. 

The general spatial ability to recall the landmarks in a virtual environment 
was not much different from the ability to recall them in a real world. Figure 4.1 shows 
the comparison of the general ability of landmark recognition for each participant in these 
two different environments. 
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Landmark Recognition for Each Participant 




Figure 4.1 Histogram for Landmark Recognition Test 

When we consider each individual’s spatial ability on landmark 
recognition, we saw that the participants who performed better in the real world 
performed better in the virtual world as well. Figure 4.2 shows a scatter plot on the 
performance of the participants in landmark recognition. 
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Figure 4.2 Landmark Recognition Comparison 



38 



The correlation coefficient for the comparison is calculated as 0.533 and 
this does not give a certain sign about the existence of a correlation. However, even in 
such a small group the performance comparison shows that in landmark recognition 
better navigators in a real world tended to be better in a virtual world as well. 
b. Sequence of Landmarks 

Just after selecting the landmarks that he had recognized, each participant 
was asked to put the landmarks into correct order. This order was measuring the 
participant’s route knowledge and landmark knowledge. The landmarks that the 
participant could accurately put into order helped to understand about which landmarks 
the participant was more confident. The results of a scatter plot for the participants’ 
performance in the sequence of the landmarks showed that the correlation between the 
ability to put the landmarks into correct order in a real and virtual world was not as strong 
as the correlation to recall them in these two environments. Figure 4.3 shows the scatter 
plot graph for the order of landmarks in these two environments. Figure 4.4 shows the 
histogram for the comparison of each individual’s performance in both environments. 
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Comparison of Landmark Order 
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Figure 4.3 Comparison of Landmark Order 

5. Route Knowledge 

Route knowledge is assessed by analyzing the order of landmarks in correct order 
and the number of wrong turns in the route replication task. Sequence of the landmarks 
which had been used as a measure of landmark knowledge can also be used as a measure 
of route knowledge. The general picture of the route knowledge presents less correlation 
in the performances in real and virtual worlds. 

a. Sequence of Landmarks 

The sequence of landmarks, as stated above, did not present a correlation 
as strong as landmark recognition. However, there were still matching good and poor 
performances. A histogram of performance level for each participant is seen in Figure 4.4. 
The scatter plot for the landmark sequence is seen in Figure 4.3. 
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Correct Order of Landmarks for Each Participant 




Participants 



Figure 4.4 Histogram for Landmark Sequencing Test 
In general, the performance of correct order in the real world tended to be 
mostly close to and in some cases better than the performance in a virtual world. The 
correlation coefficient for the correct order of landmarks was 0.447. 
b. Number of Wrong Turns 

The second important measure for route knowledge was the number of 
wrong turns in the route replication task. A navigator who has learned the route in an 
environment should make the correct turns when going through a route which he had 
taken before. A significant correlation could not be detected in the analysis of the number 
of wrong turns. Since there was a big difference between the performances in the real and 
virtual worlds, the errors in the virtual world were almost evenly distributed. The 
correlation coefficient for the number of wrong turns was -0.08. Figure 4.5 shows the 
scatter plot graph for the number of wrong turns in these two environments. 
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Number of Wrong Turns 
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Figure 4.5 Scatter Plot for the Number of Wrong Turns 

The participants made more wrong turns in the virtual than the real world. And the 
variance was also very little. This was probably because of the fact that the participants 
did not have enough time to get survey knowledge in the virtual environment. This fact 
prevented us from drawing conclusions on the route knowledge based on the results of 
performances in this test. The histogram below (Figure 4.6) shows the difference more 
clearly. 
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Figure 4.6 Histogram for the Number of Wrong Turns for each Participant 



c. Aggregated Route Knowledge Evaluation 

The correlation we had seen in the landmark knowledge was not seen in 
the performance measures of the route knowledge. So, we decided to aggregate the two 
measures of the route knowledge, and analyze these aggregated scores. 

When we look into the aggregation and consider the score as a number 
between 0 and 1, the performance of determining the correct turns in a virtual 
environment is obviously less than any other performance measure. Figure 4.7 shows a 
3D representation of the performance measures for the route knowledge. 
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Figure 4.7 3D Representation of the scores for the route knowledge 

Likewise, when we get an overall value for an individual’s route 
knowledge in a virtual or real environment, we get the scatter plot graph below. 

In an aggregated form, the performance of landmark recognition was given 
a value between 0 and 1. So, any participant who had recognized 4 of the 6 landmarks 
that had existed in the environment had a score of 4/6 = 0.66 in this model. Another value 
between 0 and 1 was also used for the number of wrong turns. This time, the number of 
wrong turns was divided to the maximum number of wrong that could be made. A greater 
value for this fraction would mean a poor performance, so the result from this division 
was subtracted from 1, and we still obtained a value between 0 and 1 for the performance 
level in the number of wrong turns. Thus, for an aggregated model, we had two values 
between 0 and 1 to indicate the performance level in these two tests. These two values 
were added, and we obtained a performance value in route knowledge between 0 and 2. 
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Figure 4.8 shows the slight correlation between the aggregated 
performances in virtual and real environments. In this graph, the values for each measure 
are added in the above way, and a score between 0 and 2 was obtained for each 
participant in a certain environment. 
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Figure 4.8 The Scatter Plot for Aggregated Route Knowledge Score 
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In the aggregated evaluation, the number of wrong turns has obviously affected the slight 
correlation in the sequence, and the graph we have in an aggregated form has the low 
coefficient of 0.04. The possible reasons for this effect will be discussed in the discussion 
section. 

6. Survey Knowledge 

Survey knowledge has been assessed by analyzing the results of the pointing tests 
and the maps built by the participants. Participants had built maps by putting the 
landmarks into certain points on the map. Although it had not been considered as a 
measure, the number of alternate routes tried by the participant have also been analyzed. 
Normally, survey knowledge is the hardest level of spatial knowledge, and can only be 
acquired in a longer time when compared to the route and landmark knowledges. 

a. Pointing Tests 

Four different pointing tests in three locations were conducted to measure 
survey knowledge. The results of the pointing tests have shown that there was a big 
difference between the performances in the real world and the virtual world. In addition 
to this, the range of degree of error in the real world was less than that in the virtual 
environment. Figures 4.9, 4.10, 4.11 and 4.12 indicate the scatter plot graphs for the 
comparison of these two environments for the four pointing tests. 

The first pointing test came with quite big errors in estimation of the 

direction. 
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Pointing Test 1 
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Figure 4.9 Scatter Plot graph for the first pointing test. 

In this first test, for the virtual environment, most of the participants have 
also expressed that their pointing was mostly based on estimation, and they had reported 
poor confidence levels. The correlation coefficient for the first pointing test was 0.157. 
Two participants have performed very well in both environments for the first task. But 
those participants were not confident about their survey knowledge while pointing to the 
points. 
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Pointing Test 2 
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Figure 4.10 Scatter Plot graph for the second pointing test. 

The second pointing test which had been conducted at the end point of the 
navigation gave better results in the virtual environment. The correlation coefficient for 
the second pointing test was -0.038. When one participant who performed very poorly in 
the real world was removed, the correlation coefficient became 0.474. However, no 
particular reason was observed for this participant’s big error. 
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Figure 4.1 1 Scatter Plot graph for the third pointing test. 

Results for the third pointing test were similar to the ones for the second, 
with the exception of the mentioned participant who made a big error in the real world. 
The correlation coefficient was -0.278. 

For the fourth pointing test, the errors in the virtual environment were still 
existing, but the real world performance was better than the previous pointing tests. The 
correlation coefficient for the fourth pointing test was -0.315. 
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Pointing Test 4 




Figure 4.12 Scatter Plot graph for the fourth pointing test 

When we consider the overall pointing tests, we see the same difference of 
performance and the lack of correlation. The one big error in one of the pointing tests 
affects the results in the overall performance too. This error was made by the same 
participant who made a big error in the second pointing test. Figure 4.13 shows the scatter 
plot graph for the overall evaluation of performances in pointing tests. 
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Results for All Pointing Tests 
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Figure 4.13 Scatter Plot graph for the overall performance in pointing tests 
b. Map Placement Tests 

Another way to analyze the survey knowledge is the map placement test 
results. The results in map placement indicated similarity to the pointing tests in the 
poorness of correct placements. So, the results of the map placement tests did not tell us 
anything about the existence of a correlation. 

Figure 4.14 shows a scatter plot graph for the map placement test. The 
correlation coefficient for this test was -0.15 of negative correlation. 
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Map Placement Test 
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Figure 4.14 Scatter Plot graph for the map placement test 

In the map placement tests, the group performance in the real world was 
still better than the performance in the virtual world. Figure 4.15 shows the histogram 
comparing each individual’s performance in these two distinct environments. As it’s seen 
in the histogram one participant made zero errors in the real world. 
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Distance Error in Map Placement 
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Figure 4.15 Comparison of Map Placement Test in Real and Virtual Environments 
c. Aggregated Evaluation 

An aggregated evaluation of the pointing tests and the map placement test 
gives a better idea about the survey knowledge of the participant. However, the 
aggregation does not indicate a correlation either. To aggregate these two measures, we 
have given a value between 0 and 100 to the pointing performance of the participant, 
simply by dividing his total error in degree by the highest error and multiplying the result 
by 100. Likewise, the distance error in map placement test was divided by the highest 
error and the result was multiplied by 100 to get a value between 0 and 100 for the map 
placement test. The addition of these two performance scores has provided a performance 
score for each participant. This score was a number between 0 and 200, and indicated the 
overall survey knowledge performance of an individual. The aggregated results showed 



2 200 

LU 

3 150 



c 

CO 




53 



that the performance in the real world was better than the performance in the virtual 
world for the sample group. Figure 4.16 shows the histogram for this comparison. 



Comparison of Error in VE and Real World for 
Aggregated Survey Knowledge 




□ Virtual Environment Error 
■ Real World Error 



Participants 



Figure 4.16 Comparison of Aggregated Results for Survey Knowledge 



When the results for the survey knowledge were aggregated, it was still 
not possible to find a positive correlation. The scatter plot for the correlation is shown in 
figure 4.17, and the correlation coefficient for the aggregated results was -0.104. 
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Aggregated Scores for Survey Knowledge 
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Figure 4.17 Scatter Plot Graph for the Aggregated Survey Knowledge 

The overall analysis of the survey knowledge gave little information about 
the performance and the existence of a correlation. 
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B. DISCUSSION 



1. General Discussion 

The general results of the experiment did not indicate a correlation in all of the 
performance measures. There was a difference between the types of spatial knowledge. 
Landmark knowledge, being the type acquired in a relatively short time, has shown a 
positive correlation for performance in real and virtual environments. Route knowledge 
was a type that is not acquired so easily, and the results for route knowledge indicate that 
there was a big difference between the performances in the real and virtual worlds. The 
environmental factors that might have had an effect are discussed under the sections of 
these types of spatial knowledge. It’s not possible to think of a virtual environment apart 
from the interface. The interface that has been used for the experiment also had some 
effects on the results, and these are also discussed under the following sections. 

2. Landmark Knowledge 

The landmark knowledge scores have indicated a correlation between performances in 
real and virtual environments. Since the spatial ability in landmark knowledge is mostly a 
memory process, this is reasonable. The participants who had the ability to keep certain 
landmarks in his memory could do so regardless of the environment. One participai;; \.ho 
did the best in the group performed as the best performer in both of the environments. 
Likewise, two participants who performed poorly in the real world did the same way in 
the virtual environment. However, there were two others who performed at a medium 
level in the real world, but poorly in the virtual environment. In an overall evaluation, it 
was seen that the landmark recognition in a virtual environment was harder than the 
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landmark recognition in a real world. This might be due to less frequent checks in a 
virtual environment. The participants seeing the landmark from one direction were not 
able to recognize the same landmark when they had seen it from another direction. This 
happened in a few cases for the candy machine. For landmark recognition, the verbal 
reports of the participants throughout the experiment have shown that some details are 
observed in a virtual world, but the bigger picture was generally lost. For instance, a 
participant was paying attention to the change of design in the horizontal and vertical 
faces of the stairways in the Herrmann Hall model, but was missing the Grandfather’s 
clock in a hall. Despite these differences, it can be said that there is a positive correlation 
between the landmark knowledge performance measures in the real world and virtual 
environment land navigation. 

It should also be noted that the objects we identified as landmarks may have not 
been meaningful landmarks to some of the participants. The concept of a landmark is 
always relative to the subjectivity of the individual. 

3. Route Knowledge 

The route knowledge performance measures did not give a level of confidence as 
much as landmark knowledge. We have used two measures to assess route knowledge. 
The first one was the order of landmarks, and the second one was the number of wrong 
turns in a route replication task. 
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While the correlation coefficient for the landmark sequencing was 0.447, it was -0.08 for 
the number of wrong turns. When we aggregated these two results, we have seen that the 
results in the number of wrong turns have affected the aggregation, and the coefficient 
was still as low as 0.04. 

One experimental observation that might have affected the results in the number 
of wrong turns and has been observed in the experiment was the ability of the participants 
in making checks in the navigation. While frequently looking over their shoulder and 
checking locations in the real world was quite simple, it was not that easy in the virtual 
environment. Since learning a route is a memory process, the checks made in the real 
world have probably helped this memory process. One other reason for the difference 
might be the fact that there were more choices in the decision points in the virtual 
environment. While a participant had three or four choices to go to in a virtual world, 
there were generally only two choices in the real world navigation. Thus the scores in the 
real world were less than the ones in the virtual world and were closer to each other. It’s 
also known that the order of development for the types of spatial knowledge acquisition is 
landmark knowledge, route knowledge, and survey knowledge. Thus, the learning of 
route knowledge itself requires more time than landmark knowledge. 

4. Survey Knowledge 

The results of the tests on survey knowledge provided very poor information. 
Since the survey knowledge is the type of spatial knowledge which requires the most 
amount of time, it is understandable that the participants were not so successful in 
forming a mental map of the buildings. However, we still asked them to provide us with 
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the answers even if those were just an estimate of the location. In these estimates, 
coincidences might have caused some of them to show good performance, and the results 
should be evaluated from this perspective. Obviously, the results did not show any 
correlation in the pointing tests and the map placements. Especially, in the pointing tests, 
there was a big difference between the results of the virtual environment and the real 
world. The real world results were much more accurate than the results of the virtual 
environment. One reason for this might be the locations of the starting point, Point A, and 
the end point. In addition to this, the way the pointing test is conducted in a virtual 
environment might have also affected the results. In a virtual environment test, since 
people do not turn with their whole body, they don’t have full confidence in pointing. 
This result is similar to the results of the study made by Peterson et al. In that study, the 
participants’ ability to form a mental map and to use it to find alternate routes was 
significantly better with a full-body controller than a joystick. (PETERSON et al, 1998) 
The observed behavior of the participants’ frequently changing position in a pointing test 
was a sign of this. While the participants performed the pointing in a real world just by 
directing the arrow, in a virtual environment they were turning their body a few times to 
make sure they face the location they intended. 

All the coefficients for the pointing tests were close to zero. It is therefore unwise 
to draw conclusions in favor of a correlation or lack of a correlation from these results. In 
the process of travel, pervious studies have shown that pointing as a traveling technique is 
more advantageous than gaze-directed steering, and this helped us to rely on a more 
comfortable way of traveling. However, it’s also been stated in previous studies that, it 
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requires more time to become an expert to be able to perform this test (BOWMAN et al. 
1997). 

In the map placement test, there were still more errors in the virtual environment. 
The fact that the participants performed relatively well in the real world for these two 
tests indicate that there was an obvious difference of performance in survey knowledge. 

Since the map placement test is also an indication of survey knowledge, it was not 
formed so easily. Having toured in the environment for a short time, the participants were 
not confident while placing the landmarks into correct locations. So, the results of the 
map placement tests did not tell us anything about the existence of a correlation. 

5. Interface Issues 

The interface for the experiment can always be questioned. The Flybox that we 
have used gave the freedom to move in all directions, but did not provide the ability to 
look around without turning the entire body. Although it may seem to developers that 
there’s no need for such a motion, this behavior was frequently used by the participants in 
the real world. Throughout navigation, the participants were frequently looking at the 
places they walked through and checking the points. This would bring an extra burden in 
a virtual environment, and the participants used this very rarely. The lack of multimodal 
stimulation while turning the body was another disadvantage of the interface. The 
participants were not able to get the sense of the direction they were turning. They were 
using only visual cues, and this largely affected their performance in the pointing tests. 
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V. CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

1. General Conclusion 

The thesis experiments studied the existence of a correlation between performance 
measures in the real world and virtual environment land navigation tasks. Participants of 
the experiment navigated in a virtual and a real world building. For each participant, his 
performance was assessed through a set of performance measures in the three types 
spatial knowledge. The results of the experiments were then analyzed in order to 
understand the existence of such a correlation, and it’s level in different measures. The 
following conclusions are drawn from the qualitative and quantitative results that have 
been presented in the previous chapters. 

a. Subjects who had the ability to keep certain landmarks in their 
memory in a virtual environment had the ability to keep landmarks in their memory in a 
real world as well. 

b. Types of spatial knowledge are formed through time, and the 
survey knowledge is the type of spatial knowledge that takes the most amount of time to 
form. 

c. The interface in a virtual environment becomes much more 
important when directional orientation is involved. 
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2 . 



Landmark Knowledge 



The results suggest that there is a positive correlation between the performance 
measures in landmark knowledge in a virtual environment and a real world. A landmark 
recognition in a virtual environment can give an idea about the performance of the same 
individual in a real world. Although not as strong as the landmark recognition, this 
correlation is seen in the sequencing of the landmarks. 

3. Route Knowledge 

The correlation in landmark knowledge was not seen in route knowledge. This 
study did not provide evidence that route knowledge performance of an individual in a 
virtual environment can give an indication about the performance of the same individual 
in a real world. 

4. Survey Knowledge 

A correlation was not seen on survey knowledge either. With the relatively very 
poor performance of the participants and the short time period spent in each environment, 
this study also supported the fact that survey knowledge is the type of spatial knowledge 
acquired over time. 
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B. RECOMMENDATIONS 



1. Different Interfaces 

This study can be replicated with different interfaces to assess the effect of the 
interfaces on the results, and further comparison can be investigated with these interfaces. 
Interfaces taking kinesthetic approaches into account will be a helpful in a better 
evaluation of the virtual environments. 

2. Different Environments 

The correlation can also be analyzed for a natural terrain outdoor environment. 
This study has mainly focused on the spatial knowledge acquisition in a building type 
environment. However, the acquisition of spatial knowledge in an outdoor environment, 
such as a forest, has its own features. Further studies can be made to investigate a 
correlation in these environments. 

3. Bigger Sample Size 

This study suggests that there may not be a correlation between the performance 
measures in real and virtual environments. A further study can be made with a bigger 
sample size. Researchers of navigation in virtual environments need to know if the lack 
of correlation is significant. 
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APPENDIX A. EXPERIMENT OUTLINE 



1) In Brief /Consent Form 

a) Time - 5 Min. 

b) Location-MO VES graphics Lab. 

c) OIC - Lt. Saltuk B. Karahan 

d) Materials -Consent Form, Privacy Act Statement , Minimal Risk Consent 
Form, pen, Briefing Scripts 

2) Interface Familiarization (VE Only) 

a) Time - 5 Min. 

b) Location- Graphics Lab. 

c) OIC - Lt. Saltuk B. Karahan 

d) Materials - SGI Machine, Herrmann Hall Trial Model, Flybox, Flybox 
Instructions, Virtual Environment Briefing Script 

3) Testing in Virtual Environment 

a) Time - Varies 

b) Location- Graphics Lab. 

c) OIC - Lt. Saltuk B. Karahan 

d) Materials - SGI Machine, Herrmann Hall Trial Model, Flybox, Flybox 
Instructions, Virtual Environment Briefing Script, Landmark Pictures, 
Building Maps, pencils. Footprint pictures of Landmarks 
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e) Tasks: 



(1) Task 1. Walking from the Starting point to the End Point 
(Perform the pointing tests on route) 

(2) Task 2. Route Replication (Measure the number of wrong turns 
on route, perform landmark recognition and landmark 
sequencing tests at the end.) 

(3) Task 3. Free Exploration 

(4) Task 4. Finding Alternate Routes 
4) Testing in Real World 

a) Time - Varies 

b) Location- Middle East School 

c) OIC - Saltuk B. Karahan 

d) Materials -Real World Briefing Scripts, Compass, Landmark Pictures, 
Building Maps, pencils, Footprint pictures of Landmarks, Pointing Board 

e) Tasks: 

(1) Task 1. Walking from the Starting point to the End Point 
(Perform the pointing tests on route by using the pointing board) 

(2) Task 2. Route Replication (Measure the number of wrong turns 
on route, perform landmark recognition and landmark 
sequencing tests at the end.) 

(3) Task 3. Free Exploration (Lead the participant to the starting 
point before the end of the allowed time) 
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(4) Task 4. Finding Alternate Routes 
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APPENDIX B. TASK LISTING 



Task 1. Move from starting point to the end point along designated route. (Measure 
the bearing from the first control point (point A) the starting point at point A. Measure the bearings 
from the end point to point A and the starting point at the ending point.) 

Task 2. Move from the end point to the starting point following the route in 
the first task in reverse order. (Measure the number of wrong turns and the locations 
for those wrong turns on route. Measure the number of landmarks correctly recognized 
and in correct order at the end of the route replication) 

Task 3. Explore the environment yourself for 10 minutes. (Record the routes 
the participant takes, perform map placement test at the end of the free exploration) 

Task 4. Find the alternate routes from the second control point (the blocked 
location) to the end point. (Record the alternate routes the participant takes) 
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APPENDIX C. BRIEFING SCRIPTS 



BRIEFING SCRIPTS FOR REAL BUILDING 

IN BRIEFING 

Welcome to the Naval Postgraduate School’s MOVES Department. My name is 
Lt. Bugra Karahan. Thank you for participating in this experiment. This experiment deals 
with navigation in Building Type environment. 

This experiment does not test your intelligence or performance in this type of an 
environment. It rather tests the measures that have been used for VE Building Type land 
navigation. Your performance will be used only for research purposes, and it will not be 
used in you records. Prior to starting the experiment you’ll be asked to read and sign a 
series of consent forms. Our experiment will take approximately 45 minutes. Upon 
completion of the task, you’ll be given a short debriefing 

If you don’t have any question, please read and sign this consent form. 

EXPERIMENT BRIEFING 

Meet the participant in the Graphics Lab. 

Take the participant to the Hachiya Hall in DLI. 

“As I have expressed, it’ll be approximately 45 minutes experiment. We will have 
four different tasks: a route following, a route replication, free exploration and finding 
alternate routes. At the beginning of each task you’ll have a brief description of the task. 
When you reach to the end point in the last task, it’ll be the end of the experiment too. 

If you don’t have any questions we can start the experiment.” 

COURSE BRIEFING 

TASK 1: STARTPOINT TO ENDPOINT. 

Take the participant to the side door of the building 623(next to room 164). 

Task: “As the requirement of the first task I’ll ask you to follow a predetermined 
route in this building. In this phase, what you’re expected to do is just to follow the verbal 
instructions to make the necessary turns in the spots I tell. We’ll have two tests on this 
first task. This is the starting point of the building. Your first task is to follow the verbal 
instructions for your navigation.” 

(Throughout the navigation, note down the number of times the participant turns 
his head to look at where he came from) 

(At the start point) -Please tum left at this spot and, go up the stairs. 

(At the room 153) - Please go straight ahead. 

(At the room 238) - Please tum right and go straight ahead on this hallway. 
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(When he reaches 232) - Please turn right to the EXIT sign. 

( When he passes the door to the stairways) - Please go down the stairways. 

(When he goes one floor down) - Please turn left. 

(When he reaches the stairway in a few meters) - Please go out of the stairway. 

(When he comes out of the stairway on the first floor of Building 621 ) - “Now 
we’ll perform a pointing task. Now face any direction you want, and make your heading 
stable. And use this marker to direct the start point.” (Note down the participant’s 
heading by using the compass, and write down the angle he directs as well.) “Thank you, 
now Please turn right, and go straight ahead on this hallway.” 

(When he comes to 123B ) “ Please turn left and go up the stairs.” 

(When he reaches the Building 619, Rm 210) - Please turn right, and go straight 
ahead on this hallway. 

(When he reaches the end of the hallway, Rm 201 ) - Please turn left to the 
stairways. 

(When he passes the door to the stairways) - Please go up the stairs. 

(When he reaches the third floor of Building 619) - Please go out of the stairways. 

(When he goes out of the stairways, and sees room 301 ) - Please turn right and go 
straight ahead on this hallway. 

(When he reaches the endpoint, Room 306) - “Please stop here. Now we’ll 
perform a second pointing task. Now face any direction you want, and make your heading 
stable. And use this marker to direct the start point. .” (Note down the participant’s 
heading by usitig the compass, and write down the angle he directs as well.) Now we’ll 
perform a third pointing task. Please face any direction you want, and make your heading 
stable. And use this marker to direct the spot where we made the first pointing task. .” 
(Note down the participant’s heading by using the compass, and write down the angle he 
directs as well.) “Thank you, this is the end of the first task” 



TASK 2: ENDPOINT TO STARTPOINT (ROUTE REPLICATION). 

Task: 

“Now, we’ll perform the second task. The second task is just to follow the exact 
route back to the starting point. At any point, if you make a wrong turn, I will not make 
any comment, but as soon as you start proceeding on that route, I’ll show you the right 
turn, and will ask you to continue from then on. 

If you don’t have any questions, now please follow the route back to the starting 

point. 



(On this task, write down all the wrong turns made by the subject and their 
locations. Use the Map 1 for this task. And throughout the navigation, note down the 
number of times the participant turns his head to look at where he came from) 

(At the end of the task.) "Thank you. This is the end of the second task. Now I’ll 
show you 10 landmarks and some of them have been on your route. Please tell me, which 
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of these landmarks you have seen on your route. ” (Note the number of correct 
landmarks) 

“Now, please put the landmarks into order from the start point to the end point. ” 
(Write down the order he has put the landmarks) 

TASK 3. FREE EXPLORATION 

Task: 

“Now, we’ll have 10 minutes of free exploration. On this free exploration. I’ll not 
help you in any way, but will just follow you on the path. When you get close to the end 
of the allowed time. I’ll lead you to the starting point. Please walk in a regular walking 
pace, and do not feel rushed. Please tell me when you’re ready for a free exploration of 
the environment.” 

( When the participant tells that he is ready, start the free exploration and the time 
as well. On the route, draw the path that the participant follows on a different map for 
further analysis. ) 

TASK 4. ALTERNATE ROUTES 

Task: 

“Now, we’ll perform the last task. Before proceeding, please point the end point. 
(Note down the participant’s heading by using the compass, and write down the angle he 
directs as well.) ‘Thank you. Now, please proceed to the end point.” 

(When he comes to the passage to the building 621) “This passage is blocked due 
to construction. Now, please show the end point, and please tell me the alternate route 
that you think of.” (Draw the alternate route on a map, but do not show that map to the 
participant. Make sure that the route you drew is exactly the same as what the participant 
told. If the route goes out of boundaries, pretend to be drawing, but just cut drawing at 
that point.) “Now, please move on your alternate route.” (If he ever wants to change the 
route, remind that this is not the alternate route he told, and ask why he has changed the 
route. If he just understood that the route he told was wrong, this is acceptable, but note 
the incident.) 

‘Thank you, we have reached at the end point. Now please place the landmarks 
you have seen on this white-board.” 

(At the end of the task, give the participant a white-board showing the exterior 
lines of the building, and make him place the landmarks that he correctly recognized on 
appropriate points.) 

“This is the end of our experiment in the Real Building. Thank you for 
participating in this experiment. I’ll debrief you on the details of this experiment after the 
completion of the whole experiment. ’’ 
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BRIEFING SCRIPTS FOR VIRTUAL BUILDING 



IN BRIEFING 

Welcome to the Naval Postgraduate School’s MOVES Department. My name is 
Lt. Bugra Karahan. Thank you for participating in this experiment. This experiment deals 
with navigation in Building Type environment. 

This experiment does not test your intelligence or performance in this type of an 
environment. It rather tests the measures that have been used for VE Building Type land 
navigation. Your performance will be used only for research purposes, and it will not be 
used in you records. Prior to starting the experiment you’ll be asked to read and sign a 
series of consent forms. Our experiment will take approximately 45 minutes. Upon 
completion of the task, you’ll be given a short debriefing 

If you don’t have any question, please read and sign this consent form. 

EXPERIMENT BRIEFING 

Meet the participant at the Main Gate. 

Bring the participant to the Graphics Lab. 

“As I have expressed, it’ll be approximately 45 minutes experiment. You’ll first 
get familiarized with the VE interface that we use for this experiment. At any phase of 
this familiarization, please feel free to ask any question. After the familiarization phase, 
you’ll have four tasks to perform in the virtual environment. Prior to each task, I’ll give 
the instructions for that specific task. When you reach to the end point in the last task, it’ll 
be the end of the experiment too. 

If you don’t have any questions we can start the experiment.” 

COURSE BRIEFING 

Brief the participant on familiarization. 

“In front of you is a three screen monitor, which you’ll use throughout your 
navigation in the experiment building. You’ll use this flybox to move in the virtual 
environment. The flybox gives you the capability of moving forward, moving backward, 
turning right, tuning left, looking down and looking up. The speed is controlled as on or 
off, from this button on the keyboard. 

If you don’t have any questions, you can try the flybox for five minutes to 
familiarize with the interface.” 



TASK 1: STARTPOINT TO ENDPOINT. 

Task: “In this phase, what you’re expected to do is just to follow the verbal 
instructions to make the necessary turns in the spots I tell. We’ll have two tests on this 
first task.” 

(At the start point) -Please go down the stairs. 

(At the base floor) - Please go straight ahead. 

(When he reaches the elevator) - Please turn right. 

(When he reaches the end of the mailboxes) - Please turn left. 

(When he gets out of the mailboxes) - Please turn left and go straight ahead. 

(When he reaches the end of the hallway) - Please turn left. 
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(When he reaches the stairway in a few meters) - Please turn left, and use the 
stairway. 

( When he comes out of the stairway on the first floor) - “Now we’ll perform a 
pointing task. Now face any direction you want, and make your heading stable. And use 
this marker to direct the start point.” (Hit the key P on the keyboard to get the exact 
heading. Note the degree that the subject shows.) “Thank you, now Please turn right, and 
go straight ahead.” 

(When he comes to the elevators) “ Please turn right and use the stairs you see.” 

(When he climbs the first part of the stairs) - Please turn left, and use the stairs 
up. 

(When he reaches the mezzanine) - “Please turn left, and go straight ahead.” 

(When he reaches the end of way) - “Please turn left, and go straight ahead.” 

(When he reaches the end of way) - “Please turn right, and go straight ahead.” 

(When he reaches the endpoint) - “Please stop here. Now we’ll perform a second 
pointing task. Now face any direction you want, and make your heading stable. And use 
this marker to direct the start point. .” (Hit the key P on the keyboard to get the exact 
heading. Note the degree that the subject shows.) Now we’ll perform a third pointing 
task. Please face any direction you want, and make your heading stable. And use this 
marker to direct the spot where we made the first pointing test. .” (Hit the key P on the 
keyboard to get the exact heading. Note the degree that the subject shows.) “Thank you, 
this is the end of the first task” 

TASK 2: ENDPOINT TO STARTPOINT (ROUTE REPUCATION). 

“Now, we’ll perform the second task. The second task is just to follow the exact 
route back to the starting point. At any point, if you make a wrong turn, I will not make 
any comment, but as soon as you start proceeding on that route, I’ 11 show you the right 
turn, and will ask you to continue from then on. 

If you don’t have any questions, now please follow the route back to the start 

point. 



(On this task, write down all the wrong turns made by the subject their locations. 
Use the Map 1 for this task. ) 

(At the end of the task.) “Thank you. This is the end of the second task. Now I’ll 
show you 10 landmarks and some of them have been on your route. Please tell me, which 
of these landmarks you have seen on your route.” (Note the number of correct 
landmarks) 

“Now, please put the landmarks into order from the start point to the end point. ” 
(Write down the order he has put the landmarks) 
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TASK 3. FREE EXPLORATION 

“Now, we’ll have 10 minutes of free exploration. On this free exploration, I’ll not 
help you in any way, but will just follow you on the path. When you complete the allowed 
amount of time, I’ll exit the program. Please tell me when you’re ready for a free 
exploration of the environment.” 

( When the participant tells that he is ready, start the free exploration and the time 
as well. On the route, draw the path that the participant follows on a different map for 
further analysis. ) 

TASK 4. ALTERNATE ROUTES 

“Now, we’ll perform the last task. Before proceeding, please point the end point. 
(Hit the key P on the keyboard to get the exact heading. Note the degree that the subject 
shows.) “Thank you. Now, please proceed to the end point. Along this route I’ll just 
follow you and.” 

( When he comes to the stairways) “This stairway is blocked due to construction. 
Now, please show the end point, and please tell me the alternate route that you think of.” 
i (Draw 

the alternate route on a map, but do not show that map to the participant. Make 
sure that the route you drew is exactly the same as what the participant told. If the route 
goes out of boundaries, pretend to be drawing, but just cut drawing at that point.) “Now, 
please move on your alternate route.” (If he ever wants to change the route, remind that 
this is not the alternate route he told, and ask why he has changed the route. If he just 
understood that the route he told was wrong, this is acceptable, but note the incident.) 

“Thank you, we have reached at the end point. Now please place the landmarks 
you have seen on this white-board.” 

(At the end of the task, give the participant a white-board, and make him place the 
landmarks on appropriate points.) 

“This is the end of our experiment in Virtual Environment. Thank you for 
participating in this experiment. I’ll debrief you on the details of this experiment after the 
completion of the entire experiment. ” 



76 



APPENDIX D. CONSENT FORMS 

NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 

MINIMAL RISK CONSENT STATEMENT 

Subject: VOLUNTARY CONSENT TO BE A RESEARCH PARTICIPANT IN : 
Navigation in Virtual Environments and Real Buildings 

I have read, understood and been provided “Information for Participants” that 
provides the details of the below acknowledgements. 

I understand that this project involves research. An explanation of the purposes of 
the research, a description of the procedures to be used, identification of experimental 
procedures, and the extended duration of my participation have been provided to me. 

I understand that this project does not involve more than minimal risk. I have been 
informed of any reasonably foreseeable risks or discomforts to me. 

I have been informed of any benefits to me or to others that may reasonably be 
expected from the research. 

I have signed a statement describing the extent to which confidentiality of records 
identifying me will be maintained. 

I have been informed of any compensation and/or medical treatments available if 
injury occurs and is so, what they consist of, or where further information may be 
obtained. 

I understand that my participation in this project is voluntary; refusal to participate 
will involve no penalty or loss of benefits to which I’m otherwise entitled. I also 
understand that I may discontinue participation at any time without penalty or loss of 
benefits to which I am otherwise entitled. 

I understand that the individual to contact should I need answers to pertinent 
questions about the research is Rudy Darken, Ph. D., Principal Investigator, and about my 
rights as a research subject or concerning a research related injury is the Modeling Virtual 
Environments and Simulation Chairman. A full and responsive discussion of the elements 
of this project and my consent as taken place. 

Medical Monitor: Flight Surgeon, Naval Postgraduate School 
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Signature of Principal Investigator 



Date 



Signature of Volunteer 



Date 



Signature of Witness 



Date 



NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 



MINIMAL RISK CONSENT STATEMENT 

Subject: VOLUNTARY CONSENT TO BE A RESEARCH PARTICIPANT IN : 
Navigation in Virtual Environments and Real Buildings 

I have read, understood and been provided “Information for Participants” that 
provides the details of the below acknowledgements. 

I understand that this project involves research. An explanation of the purposes of 
the research, a description of the procedures to be used, identification of experimental 
procedures, and the extended duration of my participation have been provided to me. 

I understand that this project does not involve more than minimal risk. I have been 
informed of any reasonably foreseeable risks or discomforts to me. 

I have been informed of any benefits to me or to others that may reasonably be 
expected from the research. 

I have signed a statement describing the extent to which confidentiality of records 
identifying me will be maintained. 

I have been informed of any compensation and/or medical treatments available if 
injury occurs and is so, what they consist of, or where further information may be 
obtained. 

I understand that my participation in this project is voluntary; refusal to participate 
will involve no penalty or loss of benefits to which I’m otherwise entitled. I also 
understand that I may discontinue participation at any time without penalty or loss of 
benefits to which I am otherwise entitled. 

I understand that the individual to contact should I need answers to pertinent 
questions about the research is Rudy Darken, Ph. D., Principal Investigator, and about my 
rights as a research subject or concerning a research related injury is the Modeling Virtual 
Environments and Simulation Chairman. A full and responsive discussion of the elements 
of this project and my consent as taken place. 



Signature of Principal Investigator 



Date 



Signature of Volunteer 



Date 



Signature of Witness 



Date 
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NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 



PRIVACY ACT STATEMENT 

Authority: Naval Instruction 

Purpose: Spatial Cognition information will be collected to enhance knowledge, 
or to develop tests, procedures, and equipment to improve the development of virtual 
environments. 

Use: Spatial Cognition information will be used for statistical analysis by the 
Departments of the Navy and Defense, and other U.S. Government agencies, provided 
this use is compatible with the purpose for which the information was collected. Use of 
the information may be granted to legitimate non-govemment agencies or individuals by 
the Naval Postgraduate School in accordance with the provisions of the Freedom of 
Information Act. 

Disclosure/Confidentiality: 

I have been assured that my privacy will be safeguarded. I will be assigned a 
control or code number which thereafter will be the only identifying entry on any of the 
research records. The principal investigator will maintain the cross-reference between 
name and control number. It will be decoded only when beneficial to me or if some 
circumstances, which is not apparent at this time, would make it clear that decoding 
would enhance the value of the research data. In all cases, the provisions of the Privacy 
Act Statement will be honored. 

I understand that a record of the information contained in this Consent Statement 
or derived from the experiment described here in will be retained permanently at the 
Naval Postgraduate School or by higher authority. I voluntarily agree to its disclosure to 
agencies or individuals indicated in paragraph 3 and I have been informed that failure to 
agree to such disclosure may negate the purpose for which the experiment was conducted. 

I also understand that disclosure of the requested information, including my Social 
Security Number, is voluntary. 



Signature of Volunteer Print Name, Grade/Rank (if applicable) DOB SSN Date 



Signature of Witness 



Date 



APPENDIX E. BUILDING MAPS AND ROUTES 



Herrmann Hall First Floor 
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Herrmann Hall Second Floor 
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BUILDING 220 
MAIN FLOOR 



Herrmann Hall Third Floor 
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Real Building First Floor 
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Real Building Second Floor 
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Real Building Third Floor 
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APPENDIX F. DATA COLLECTION WORKSHEETS 
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APPENDIX F. DATA COLLECTION WORKSHEETS 
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APPENDIX F. DATA COLLECTION WORKSHEETS 
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